Skip to content

TT-16964 - remove sbom job from release workflow#962

Merged
bsten-tyk merged 5 commits intomasterfrom
fix/TT-16964/remove-sbom-job
Apr 14, 2026
Merged

TT-16964 - remove sbom job from release workflow#962
bsten-tyk merged 5 commits intomasterfrom
fix/TT-16964/remove-sbom-job

Conversation

@bsten-tyk
Copy link
Copy Markdown
Contributor

@bsten-tyk bsten-tyk commented Apr 14, 2026

Description

This PR fixes three CI/CD issues introduced by commit 339740c ("security: pin actions to SHA, harden CI permissions, pin Docker images" #956) that have broken both the ci-test.yml and release.yml workflows on master since April 1, 2026.

Changes:

  1. Remove sbom job from release.yml -- The sbom job calls a reusable workflow that requests id-token: write, but the workflow-level permissions block only grants contents: read. This caused all release workflow runs on master to fail with startup_failure.

  2. Fix goimports install in ci-test.yml -- The security hardening changed go install to go run for goimports. go run executes the tool once but does not install the binary to PATH, causing ci-test.sh to fail with "goimports: command not found". Restored to go install with a pinned version (@v0.33.0).

  3. Fix multi-platform manifest digests in Dockerfile.distroless -- Base images were pinned to single-platform (amd64-only) SHA256 digests instead of multi-platform manifest list digests. This broke arm64 and s390x Docker builds because dpkg rejected architecture mismatches. Replaced with manifest list digests that cover all required platforms.

Related Issue

TT-16964

Motivation and Context

Both CI workflows (ci-test.yml and release.yml) have been broken on master since April 1, 2026 due to unintended side effects from the security hardening commit. No successful release workflow run has occurred since March 26. This PR restores CI functionality while preserving the security intent (pinned versions and digests).

How This Has Been Tested

  • The sbom permission error is a workflow validation failure at parse time -- removing the job eliminates it.
  • The goimports fix was verified by analyzing the CI failure logs showing "command not found" at ci-test.sh:33.
  • The Dockerfile digest fix was verified using docker buildx imagetools inspect to confirm manifest list digests cover amd64, arm64, and s390x platforms.

Screenshots (if appropriate)

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • Make sure you are requesting to pull a topic/feature/bugfix branch (right side).
  • Make sure you are making a pull request against the master branch (left side).
  • My change requires a change to the documentation.
  • All new and existing tests passed.
  • Check your code additions will not fail linting checks:
    • go fmt -s
    • go vet

The sbom job was requesting 'id-token: write' permission but the
workflow-level permissions only grant 'id-token: none', causing
the workflow validation to fail with:

  Error calling workflow '...sbom.yaml': The nested job 'sbom' is
  requesting 'id-token: write', but is only allowed 'id-token: none'.

Removing the sbom job resolves the validation error.

Refs: TT-16964
Signed-off-by: bsten-tyk <221599321+bsten-tyk@users.noreply.github.com>
@bsten-tyk bsten-tyk requested a review from a team as a code owner April 14, 2026 08:19
@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Apr 14, 2026

Security Issues (4)

Severity Location Issue
🔴 Critical .github/workflows/release.yml:516-522
The Software Bill of Materials (SBOM) generation job has been removed from the release workflow. SBOMs are a critical component of software supply chain security, providing visibility into dependencies and potential vulnerabilities. Removing this step significantly degrades the security posture of the release process and should be considered a critical regression.
💡 SuggestionThe underlying permissions issue that caused the job to fail should be resolved instead of removing the job entirely. The `sbom` job requires `id-token: write` permission, which can be granted specifically to this job by adding a `permissions` block within the job definition. This restores the security control while fixing the workflow.
🟠 Error ci/Dockerfile.distroless:3
The base Docker image `debian:trixie-slim` is no longer pinned to a specific SHA256 digest. Using a floating tag (like `trixie-slim`) makes the build non-reproducible and vulnerable to tag mutation attacks, where the tag could be maliciously updated to point to a compromised image.
💡 SuggestionTo ensure build integrity and security, pin the base image to a multi-platform manifest list digest. You can find the correct digest using a command like `docker buildx imagetools inspect debian:trixie-slim` and then append it to the image name (e.g., `debian:trixie-slim@sha256:...`).
🟠 Error ci/Dockerfile.distroless:12
The base Docker image `gcr.io/distroless/static-debian12:nonroot` is no longer pinned to a specific SHA256 digest. Using a floating tag makes the build non-reproducible and vulnerable to tag mutation attacks, where the tag could be maliciously updated to point to a compromised image.
💡 SuggestionTo ensure build integrity and security, pin the base image to a multi-platform manifest list digest. You can find the correct digest using a command like `docker buildx imagetools inspect gcr.io/distroless/static-debian12:nonroot` and then append it to the image name (e.g., `gcr.io/distroless/static-debian12:nonroot@sha256:...`).
🟡 Warning .github/workflows/ci-test.yml:67
The `goimports` command was changed from `go run ... -l .` to `go install ...`. The original command checked for formatting issues and would fail the build if any were found. The new command only installs the tool, effectively disabling the formatting check at this stage of the CI pipeline.
💡 SuggestionRestore the formatting check. After installing the tool, add a step to run it with the `-l` flag and fail the job if it produces any output. This ensures that code formatting is consistently enforced.
🔧 Suggested Fix
run: |
  go install golang.org/x/tools/cmd/goimports@v0.33.0
  if [ -n "$(goimports -l .)" ]; then
    echo "goimports found files that need formatting:"
    goimports -l .
    exit 1
  fi

Security Issues (4)

Severity Location Issue
🔴 Critical .github/workflows/release.yml:516-522
The Software Bill of Materials (SBOM) generation job has been removed from the release workflow. SBOMs are a critical component of software supply chain security, providing visibility into dependencies and potential vulnerabilities. Removing this step significantly degrades the security posture of the release process and should be considered a critical regression.
💡 SuggestionThe underlying permissions issue that caused the job to fail should be resolved instead of removing the job entirely. The `sbom` job requires `id-token: write` permission, which can be granted specifically to this job by adding a `permissions` block within the job definition. This restores the security control while fixing the workflow.
🟠 Error ci/Dockerfile.distroless:3
The base Docker image `debian:trixie-slim` is no longer pinned to a specific SHA256 digest. Using a floating tag (like `trixie-slim`) makes the build non-reproducible and vulnerable to tag mutation attacks, where the tag could be maliciously updated to point to a compromised image.
💡 SuggestionTo ensure build integrity and security, pin the base image to a multi-platform manifest list digest. You can find the correct digest using a command like `docker buildx imagetools inspect debian:trixie-slim` and then append it to the image name (e.g., `debian:trixie-slim@sha256:...`).
🟠 Error ci/Dockerfile.distroless:12
The base Docker image `gcr.io/distroless/static-debian12:nonroot` is no longer pinned to a specific SHA256 digest. Using a floating tag makes the build non-reproducible and vulnerable to tag mutation attacks, where the tag could be maliciously updated to point to a compromised image.
💡 SuggestionTo ensure build integrity and security, pin the base image to a multi-platform manifest list digest. You can find the correct digest using a command like `docker buildx imagetools inspect gcr.io/distroless/static-debian12:nonroot` and then append it to the image name (e.g., `gcr.io/distroless/static-debian12:nonroot@sha256:...`).
🟡 Warning .github/workflows/ci-test.yml:67
The `goimports` command was changed from `go run ... -l .` to `go install ...`. The original command checked for formatting issues and would fail the build if any were found. The new command only installs the tool, effectively disabling the formatting check at this stage of the CI pipeline.
💡 SuggestionRestore the formatting check. After installing the tool, add a step to run it with the `-l` flag and fail the job if it produces any output. This ensures that code formatting is consistently enforced.
🔧 Suggested Fix
run: |
  go install golang.org/x/tools/cmd/goimports@v0.33.0
  if [ -n "$(goimports -l .)" ]; then
    echo "goimports found files that need formatting:"
    goimports -l .
    exit 1
  fi
\n\n ### Architecture Issues (3)
Severity Location Issue
🟠 Error .github/workflows/ci-test.yml:67
The `goimports` hygiene check has been effectively disabled. The command was changed from `go run ... -l .`, which executes the check, to `go install ...`, which only installs the tool. This silently removes a code quality gate from the CI pipeline, which is an architectural regression.
💡 SuggestionThe step should be modified to both install `goimports` (to make it available for later scripts) and execute the formatting check. The check should fail the job if any files need reformatting. For example: ```yaml run: | go install golang.org/x/tools/cmd/goimports@v0.33.0 if [ -n "$(goimports -l .)" ]; then echo "goimports found formatting issues" exit 1 fi ```
🟠 Error .github/workflows/release.yml:516-522
The `sbom` job has been removed to resolve a workflow permission error. This eliminates Software Bill of Materials (SBOM) generation, which is a critical component for supply chain security. Removing this security feature to fix a configuration issue is a significant architectural regression.
💡 SuggestionInstead of removing the job, resolve the permission error. The reusable workflow requires `id-token: write`. This permission should be granted specifically to the `sbom` job within this workflow. ```yaml sbom: needs: goreleaser uses: TykTechnologies/github-actions@d3fa208 # main permissions: id-token: write secrets: DEPDASH_URL: ${{ secrets.DEPDASH_URL }} DEPDASH_KEY: ${{ secrets.DEPDASH_KEY }} ORG_GH_TOKEN: ${{ secrets.ORG_GH_TOKEN }} ```
🟡 Warning ci/Dockerfile.distroless:3-12
The Dockerfile base images are no longer pinned by digest. While removing the previous single-platform digests fixes the multi-architecture build, it sacrifices build reproducibility and security by relying on mutable tags. The architectural best practice is to pin to a multi-platform manifest list digest.
💡 SuggestionIdentify the multi-arch manifest list digests for `debian:trixie-slim` and `gcr.io/distroless/static-debian12:nonroot` and update the `FROM` instructions to use them. This can be done using a tool like `docker buildx imagetools inspect <image>:<tag>`.

Performance Issues (1)

Severity Location Issue
🟡 Warning ci/Dockerfile.distroless:3-12
Base images in the Dockerfile are specified using floating tags (`debian:trixie-slim` on line 3 and `gcr.io/distroless/static-debian12:nonroot` on line 12) instead of pinned digests. While this change was necessary to enable multi-architecture builds, it can degrade build performance. Using floating tags can lead to build cache misses if the base image is updated, as Docker will pull the new version and rebuild subsequent layers. This increases build times and makes builds non-deterministic.
💡 SuggestionTo improve build performance and ensure reproducibility, pin the base images to their multi-platform manifest list digest. This provides the required multi-architecture support while guaranteeing that the build cache is leveraged effectively. You can obtain the correct digest by running `docker buildx imagetools inspect <image>:<tag>`.

Quality Issues (1)

Severity Location Issue
🟠 Error .github/workflows/ci-test.yml:67
The `goimports` hygiene check was effectively removed. The original command `go run ... goimports ... -l .` would list files that need formatting. The new command `go install ...` only installs the tool but does not run any check. This removes the code formatting validation from the CI pipeline.
💡 SuggestionRestore the `goimports` check to ensure code formatting is validated. The tool should be installed and then executed with a check that fails the step if formatting issues are found. A common pattern is to check if the output of `goimports -l .` is empty.

Example fix:

go install golang.org/x/tools/cmd/goimports@v0.33.0
if [ -n &#34;$(goimports -l .)&#34; ]; then
  echo &#34;Go files are not formatted with goimports:&#34;
  goimports -l .
  exit 1
fi

Powered by Visor from Probelabs

Last updated: 2026-04-14T10:57:19.403Z | Triggered by: pr_updated | Commit: c2444a0

💡 TIP: You can chat with Visor using /visor ask <your question>

@bsten-tyk bsten-tyk changed the title fix: remove sbom job from release workflow [TT-16964] TT-16964 - remove sbom job from release workflow Apr 14, 2026
@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Apr 14, 2026

This PR addresses critical failures in the ci-test.yml and release.yml workflows that have been blocking CI and releases. It introduces three main changes: removing the sbom job to resolve a permissions error, fixing a goimports command that was failing tests, and correcting Docker base image digests to support multi-architecture builds.

While the PR successfully unblocks the pipelines, it introduces significant side effects. The removal of the sbom job eliminates a key software supply chain security artifact. Additionally, the fix to the goimports step inadvertently disables the automated code formatting check, weakening code quality enforcement.

Files Changed Analysis

  • .github/workflows/release.yml: The sbom job has been completely removed to work around a workflow-level permission error (id-token: write was required but not granted).
  • .github/workflows/ci-test.yml: The command go run ... goimports ... -l . was replaced with go install .... This fixes a "command not found" error but no longer performs the formatting check, as the -l . flags that list unformatted files have been removed.
  • .github/workflows/linter.yaml: The gotestsum dependency is updated from v1.12.1 to v1.13.0.
  • ci/Dockerfile.distroless: The base images (debian:trixie-slim and gcr.io/distroless/static-debian12:nonroot) are no longer pinned to specific SHA256 digests. This allows Docker to use multi-platform manifest lists, fixing builds for non-amd64 architectures.

Architecture & Impact Assessment

  • What this PR accomplishes: It provides an immediate fix for broken CI/CD pipelines, unblocking releases and pull request checks.
  • Key technical changes introduced:
    1. Removal of SBOM Generation: The sbom job is deleted from the release workflow.
    2. Disabling of Go Formatting Check: The goimports hygiene check is effectively disabled.
    3. Enabling Multi-Arch Docker Builds: Dockerfiles are corrected to use multi-platform base images.
  • Affected system components:
    • Release Workflow: The workflow is now functional but no longer produces a Software Bill of Materials (SBOM), which reduces visibility into software supply chain security.
    • CI Code Quality: The automated check for Go import formatting is no longer active, potentially leading to inconsistent code style.
graph TD
    subgraph "Release Workflow (Before)"
        A[goreleaser] --> B{sbom job};
        B --x|"Fails due to id-token permission error"| C[Workflow Failure];
    end

    subgraph "Release Workflow (After)"
        D[goreleaser] --> E[Release Succeeds];
        F((No SBOM Generated));
        E --> F;
    end
Loading

Scope Discovery & Context Expansion

  • The changes are localized to CI/CD configuration files, but their impact is significant. The removal of the sbom job should be treated as a temporary measure. The underlying issue is a permissions mismatch that can be resolved by granting id-token: write at the job level within release.yml, which would be a more robust and secure solution.
  • The modification to ci-test.yml represents a regression in code quality enforcement. The intended fix for the "command not found" error should be followed by a step that executes the goimports -l . check to ensure code formatting is still validated.
Metadata
  • Review Effort: 2 / 5
  • Primary Label: bug

Powered by Visor from Probelabs

Last updated: 2026-04-14T10:57:26.728Z | Triggered by: pr_updated | Commit: c2444a0

💡 TIP: You can chat with Visor using /visor ask <your question>

The security hardening commit 339740c changed 'go install' to 'go run'
for goimports, which only executes it once without installing the
binary. This caused ci-test.sh to fail with 'goimports: command not
found' since the binary was never placed on PATH.

Using 'go install' with a pinned version (@v0.33.0) satisfies both
the security requirement (pinned, not @latest) and the functional
requirement (binary available for ci-test.sh).

Refs: TT-16964
Signed-off-by: bsten-tyk <221599321+bsten-tyk@users.noreply.github.com>
gotestsum@v1.12.1 depends on golang.org/x/tools@v0.24.0 which is
incompatible with Go 1.25 (invalid array length -delta * delta in
tokeninternal.go:64). Upgrading to gotestsum@v1.13.0 pulls in
golang.org/x/tools@v0.36.0 which compiles cleanly.

Refs: TT-16964
Signed-off-by: bsten-tyk <221599321+bsten-tyk@users.noreply.github.com>
The security hardening commit 339740c pinned base images to SHA256
digests, but used single-platform (amd64-only) digests instead of
multi-platform manifest list digests. This breaks multi-arch Docker
builds because arm64/s390x build stages pull the amd64-only base image.

Replace with manifest list digests that cover all required platforms:
- debian:trixie-slim: amd64, arm64, s390x, and others
- gcr.io/distroless/static-debian12:nonroot: amd64, arm64, s390x, ppc64le

Verified via: docker buildx imagetools inspect <image>

Refs: TT-16964
Signed-off-by: bsten-tyk <221599321+bsten-tyk@users.noreply.github.com>
Comment thread ci/Dockerfile.distroless Outdated
RUN dpkg -i /${BUILD_PACKAGE_NAME}_*${TARGETARCH}.deb && rm /*.deb

FROM gcr.io/distroless/static-debian12:nonroot@sha256:5074667eecabac8ac5c5d395100a153a7b4e8426181cca36181cd019530f00c8
FROM gcr.io/distroless/static-debian12:nonroot@sha256:a9329520abc449e3b14d5bc3a6ffae065bdde0f02667fa10880c49b35c109fd1
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can add sha here, we depend on this step to get the latest debian security fix during the build time for all our releases.
If we pin to a sha here, it would break this workflow, and we'll no longer get security fixes, unless we update the image digests for every release, and all our CVE timelines/SLAs for debian fixes would break.
I think it could work with an associated workflow with dependabot or digestabot to update digests, but these PRs should still be approved, merged and picked to the right branches to get the latest updates.
Until we have such similar steps set up in practice, I don't think we can use sha pinning here, and will have to accept this risk, and trust google/debian that the upstream images pulled using tags are safe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, updated tags in c2444a0 based on your suggestion.

SHA pinning base images prevents automatic Debian security patch
updates from flowing through. Without automated digest bumping
(dependabot/digestabot), pinned digests will fall behind on CVE
fixes. Revert to unpinned tags matching the tyk gateway project
and gromit-generated templates.

Refs: TT-16964
Signed-off-by: bsten-tyk <221599321+bsten-tyk@users.noreply.github.com>
@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@bsten-tyk bsten-tyk enabled auto-merge (squash) April 14, 2026 13:35
Comment thread ci/Dockerfile.distroless
RUN dpkg -i /${BUILD_PACKAGE_NAME}_*${TARGETARCH}.deb && rm /*.deb

FROM gcr.io/distroless/static-debian12:nonroot@sha256:5074667eecabac8ac5c5d395100a153a7b4e8426181cca36181cd019530f00c8
FROM gcr.io/distroless/static-debian12:nonroot
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move to debian 13 here, as with nonroot static images, there shouldn't be no issue with bumping debian version. (No GLIBC)

@bsten-tyk bsten-tyk merged commit 93fcdd6 into master Apr 14, 2026
62 of 70 checks passed
@bsten-tyk bsten-tyk deleted the fix/TT-16964/remove-sbom-job branch April 14, 2026 14:43
@lghiur
Copy link
Copy Markdown
Collaborator

lghiur commented Apr 14, 2026

/release to release-5.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants